fix(DEPLOY-005): resolve pinned universe sources from the repo, not DATA_DIR - #116
Open
DoRmAmMu1997 wants to merge 2 commits into
Open
fix(DEPLOY-005): resolve pinned universe sources from the repo, not DATA_DIR#116DoRmAmMu1997 wants to merge 2 commits into
DoRmAmMu1997 wants to merge 2 commits into
Conversation
…ATA_DIR
`refresh_universe_files()` could not rebuild the three Hemant universes in any
deployment that sets DATA_DIR, because their pinned source lists were resolved
from UNIVERSE_DIR (which follows DATA_DIR) rather than from the code.
In the image, `COPY . .` puts the lists at /app/data/universes/, but Render and
docker-compose set DATA_DIR=/data, so the builder looked on the data volume and
raised `FileNotFoundError: Universe source CSV not found:
/data/universes/hemant_super_45.csv`. Two consequences:
* The daily-scan cron never ran. render.yaml runs
`sh -c "python -c '...refresh_universe_files()' && run_daily_scan"` on an
ephemeral filesystem with no disk at all, so the refresh raised every night
and `&&` meant the scan was skipped.
* First-deploy seeding failed. docs/operations.md tells the operator to run
that exact command in a Render Shell to populate the empty disk.
The same coupling meant a generated universe CSV was also its own input: each
refresh rewrote the file it had just read, so the working tree went dirty after
every app run and every committed row carried an absolute developer path in its
`source` column.
Split the two concerns onto separate anchors:
* inputs - `UNIVERSE_SOURCE_DIR`, anchored to PROJECT_ROOT, pinned, reviewed,
shipped in the image (new `data/universes/sources/*.csv`, extracted verbatim
from the existing `source_symbol` values - counts and ordering unchanged at
43 / 43 / 262).
* outputs - `UNIVERSE_DIR`, still following DATA_DIR onto the data volume.
Also store `source` as a repo-relative POSIX label via
`repo_relative_source_label()`, and rewrite the three committed CSVs so they no
longer embed a developer's home directory. That diff is the source column only.
The regression test runs in a subprocess with a relocated DATA_DIR, because the
faulty binding happened at import time and monkeypatching an already-imported
process would prove nothing. Verified failing before this change with the exact
production error, and passing after.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
refresh_universe_files()could not rebuild the three Hemant universes in any deployment that setsDATA_DIR.Their pinned source lists were resolved from
UNIVERSE_DIR, which followsDATA_DIR. ButCOPY . .puts those lists at/app/data/universes/, so a container withDATA_DIR=/datalooked on the data volume instead and raised:/app/data/universes/WORKDIR /app+COPY . .(Dockerfile:29,52);.dockerignorewhitelists the four tracked CSVsDEFAULT_DATA_DIR = /app/dataPROJECT_ROOT = parents[2](settings.py:27,30)DATA_DIR=/data→universe_dir = /data/universesrender.yaml:60,104;settings.py:341,122/data/universes/<key>.csvUNIVERSE_DIRTwo blast radii
render.yamlruns it assh -c "python -c '...refresh_universe_files()' && run_daily_scan"on an ephemeral filesystem with no disk at all (render.yaml:95). So/datawas empty every run, the refresh raised, and&&meant the scan was skipped. Every night.docs/operations.md:912tells the operator to run that exact command in a Render Shell to populate the empty disk.The blueprint comment shows how the gap was reasoned into existence — "the nifty/fno lists are downloaded, not baked into the image". True for NIFTY/F&O; the Hemant lists are baked in, just at a path nothing looked at. That comment is corrected here.
The fix
Separate the two concerns onto different anchors:
UNIVERSE_SOURCE_DIR, anchored toPROJECT_ROOT: pinned, reviewed, shipped inside the image. Newdata/universes/sources/*.csv, extracted verbatim from the existingsource_symbolvalues, so counts and ordering are unchanged (43 / 43 / 262, and the existing pinned-snapshot assertions on the last three tokens still hold).UNIVERSE_DIR, still followingDATA_DIRonto the data volume.This also closes a second-order smell the coupling caused: source and output being one path meant every refresh rewrote the file it had just read, so the working tree went dirty after any app run, and every committed row carried an absolute developer path in its
sourcecolumn.repo_relative_source_label()now stores a repo-relative POSIX label, and the three committed CSVs are rewritten accordingly — that diff is thesourcecolumn only.On the regression test
It runs in a subprocess with a relocated
DATA_DIR, and that is deliberate rather than lazy: the faulty binding happened at import time, so monkeypatching the environment inside an already-imported process would have proved nothing. A fresh interpreter is exactly what the Render cron anddocker composegive us.Verified failing before this change with the exact production error above, and passing after.
Verification
pytest -q --cov=... --cov-fail-under=89— 2037 passed, 1 skipped, 89.96%ruff check,compileall,bandit— cleangit diff origin/main HEAD -- constraints.txt pyproject.toml— empty (AGENTS.md §7)origin/mainmerged in and all gates re-run after PRs fix(DATA-003): stop the scan re-dirtying the candle cache it just repaired #112/fix(DATA-004): stop re-downloading stocks that listed after the window opened #114 landedCaveat, stated plainly: this was traced statically through committed files and reproduced in-process. Docker is not installed on the development machine, so the container-level run was not performed. If the Render blueprint was never applied, the defect was latent rather than live.
Docs updated: universe-management LLD (including a new design-decision row),
render.yaml,docs/operations.md,README.md.🤖 Generated with Claude Code